This paper is methodological (and experimental) in nature, providing a suite of approaches to differentially-private Bayesian linear regression. The key significance is to revisit DP linear regression in the Bayesian setting, where it is natural to consider 1) how privacy-preserving noise affects posterior estimates; 2) leverage Bayesian inference through directly modelling the noise process, to improve utility (broadly construed including in terms of calibration). The paper does a quality job of exploring how such modelling and inference could be performed based on sufficient statistic perturbation. The paper has high clarity, further adding to the potential practical impact. The main technical ideas are largely inspired by prior work such as Bernstein and Sheldon (2018)'s work on exponential families.

All reviewers consider the paper to make a solid contribution and recommend acceptance.

Linear regression is an important tool across many fields that work with sensitive human-sourced data. Significant prior work has focused on producing differentially private point estimates, which provide a privacy guarantee to individuals while still allowing modelers to draw insights from data by estimating regression coefficients. We investigate the problem of Bayesian linear regression, with the goal of computing posterior distributions that correctly quantify uncertainty given privately released statistics. We show that a naive approach that ignores the noise injected by the privacy mechanism does a poor job in realistic data settings. We then develop noise-aware methods that perform inference over the privacy mechanism and produce correct posteriors across a wide range of scenarios.

Linear regression is an important tool across many fields that work with sensitive human-sourced data. Significant prior work has focused on producing differentially private point estimates, which provide a privacy guarantee to individuals while still allowing modelers to draw insights from data by estimating regression coefficients. We investigate the problem of Bayesian linear regression, with the goal of computing posterior distributions that correctly quantify uncertainty given privately released statistics. We show that a naive approach that ignores the noise injected by the privacy mechanism does a poor job in realistic data settings. We then develop noise-aware methods that perform inference over the privacy mechanism and produce correct posteriors across a wide range of scenarios.